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Objectives: This study presents the information system for Pusan National University Hospital (PNUH), evaluates its perfor- 
mance qualitatively and conducts economic analysis. Methods: Information system for PNUH was designed by component- 
based development and developed by internet technologies. Order Communication System, Electronic Medical Record, and 
Clinical Decision Support System were newly developed. The performance of the hospital information system was qualita- 
tively evaluated based on the performance reference model in order to identify problem areas for the old system. The Infor- 
mation Economics approach was used to analyze the economic feasibility of hospital information system in order to account 
for the intangible benefits. Results: Average performance scores were 3.16 for input layer, 3.35 for process layer, and 3.57 for 
business layer. In addition, the cumulative benefit to cost ratio was 0.50 in 2011, 1.73 in 2012, 1.76 in 2013, 1.71 in 2014, and 
1.71 in 2015. The B/C ratios steadily increase as value items are added. Conclusions: While overall performance scores were 
reasonably high, doctors were less satisfied with the system, perhaps due to the weak clinical function in the systems. The 
information economics analysis demonstrated the economic profitability of the information systems if all intangible benefits 
were included. The second qualitative evaluation survey and economic analysis were proposed to evaluate the changes in per- 
formance of the new system. 
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I. Introduction 

Pusan National University Hospital (PNUH) was founded 
on November 1, 1956 as the only national hospital in Pusan 
city and established an affiliated hospital in Yangsan, a newly 
developed city on the outskirts of Pusan, in 2008. PNUH has 
1,170 beds and 2,100 staffs, treated 911,649 outpatients and 
361,246 inpatients, and performed about 16,000 operations 
in 2010. PNUH is the largest medical center in the South- 
eastern region in Korea with several specialized centers 
including regional emergency-care center, regional cancer 
center, and cardiovascular center. Recently, a trauma center, 
pulmonary center, and regional clinical trials center were es- 
tablished with special government grants. 

Information Communication Technology has been widely 
applied in hospitals worldwide. The Healthcare Informa- 
tion and Management Systems Society categorized hospital 
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information system according to eight stages [1]. Stage 0 
systems do not have medical support subsystems such as 
laboratory, pharmacy, and radiology. Stage 1 systems include 
a medical support system. Stage 2 systems include clinical 
data repository (CDR) utilizing information from the medi- 
cal support systems. Stage 3 systems include clinical decision 
support system (CDSS) integrated with CDR and nursing 
records. Stage 4 systems include a computerized physician 
order entry (CPOE) and CDSS based on the knowledge ac- 
quired from the evidence-based medicine. Stage 5 systems 
integrate the information provided by CPOE, CDSS, and 
medical support systems. Stage 6 systems include electronic 
medical record (EMR) and CDSS based on Critical Pathway 
and Clinical Practice Guidelines. Finally, State 7 systems al- 
low for inter- op erability among systems by using Continuity 
of Care Document. 82% of information systems for tertiary 
hospitals in Korea, including PNUH, belong to Stage 6 [2]. 
Recently, the Seoul National University Bundang Hospital in 
Korea became only the eighth hospital in the world and the 
first outside the United States to receive a certificate for Stage 
7 [3]. 

Healthcare systems in Korea are heavily dominated by the 
private sector, with only 78 (4%) of 1,943 hospitals being 
public in 2007 [4] . Except a few national university hospitals, 
most public hospitals suffer from inadequate facilities and 
manpower, mainly due to a lack of government support. 
While most private hospitals are increasingly introducing 
state-of-art hospital information systems (HISs) such as 
EMR and CDSS (Stage 6) in order to improve the quality of 
services, many public hospitals cannot afford such upgrades. 
To deal with this imbalance in information system (IS) 
adoption between public and private hospitals, government 
has decided to invest in developing the model for IS for pub- 
lic hospitals at PNUH. 

The performance of ISs should be evaluated from both 
qualitative and quantitative perspectives before and after 
their implementation in order to improve user satisfaction 
and justify capital investment on IS. Qualitative evaluation 
deals with the factors related to user satisfaction and im- 
provement in the business process. The Performance Refer- 
ence Model (PRM) was developed by the Office for Manage- 
ment of Budget in the United States and has been widely 
used in evaluating IS performance for government projects 
in the United States. Chung [5] applied PRM to evaluate IS 
performance for several government projects in Korea. Since 
PRM evaluates IS based on the key performance indicators 
(KPI) for each of three system layers (input, process, and 
business), it provides a comprehensive view of the overall 
IS performance. They developed PRM ver. 2.0 as a standard 
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performance evaluation tool and identified 138 KPIs for 
evaluating the performance of government IS projects. How- 
ever, many of these KPIs are not applicable to the healthcare 
setting because they were primarily developed for ISs for 
government projects unrelated to healthcare. Therefore, a 
need has arisen to identify KPIs for evaluating the perfor- 
mance of IS projects in a healthcare setting based on the 
PRM framework. 

Quantitative evaluation primarily deals with the economic 
effects of IS. Although several studies have conducted eco- 
nomic evaluations of IS projects in healthcare [6,7], most of 
these studies did not adequately account for intangible or 
indirect benefits such as savings of opportunity costs due to 
reduced waiting time. Parker et al. [8] introduced the infor- 
mation economics (IE) approach to account for intangible 
benefits by systematically analyzing intangible benefit items. 
Chae et al. [9] used IE to analyze the economic effects of 
CPOE and Kang et al. [10] also studied the economic feasi- 
bility of EMR using IE. In order to disseminate the system 
developed specifically for PNUH to other public hospitals, 
the system must be evaluated and demonstrated its econom- 
ic feasibility. 

Therefore, this study presents the Pusan Hospital Informa- 
tion System (PHIS), evaluates its performance qualitatively 
based on PRM, and conducts economic analysis by using the 
IE approach. 

II. Case Description 

1. History 

PNUH first introduced the order communication system 
(OCS), often called the CPOE system, the admission/ dis- 
charge/transfer (ADT) system, and billing systems in 1997. 
The picture archiving and communication system (PACS) 
was introduced in 2004 and EMR was introduced to the 
Yangsan PNUH in 2008. 

As shown in Figure 1, PNUH was designated as a demon- 
stration site for the new PHIS by the Ministry of Health and 
Welfare in March, 2009. A new task force was organized to 
carry out planning, identifying user requirement, and user 
training. This task force was comprised of a planning com- 
mittee (10 members), user coordinators (9 nurses, 3 infor- 
mation specialists), 5 user training instructors, a nursing in- 
formation committee (11 members), a system development 
team (52 members), and a key user team (180 members). 
A total of 1,179 person-months (average 50 persons per 
month) was devoted to the development of the PHIS. Se- 
lected members from the task force visited several hospitals 
to learn about EMR and the implementation problems ex- 
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perienced by these hospitals. Survey on user satisfaction was 
conducted in July, 2010 to elucidate the user problems and 
identify areas for improvement. The PHIS was finally opened 



on January 29, 20 1 1 . 

A number of user training sessions, seminars, and system 
rehearsals have been conducted since the development proj- 
ect was approved in March, 2009 (Table 1). A total of 14,095 
task force members have participated during this period. 



Table 1. Summary of activities for the hospital information sys- 
tem task force 



Activities Frequency 



User training 
Meeting 



Seminar 

Rehearsal and 
unit test 



Total 



No. of 
participants 



Specific 
activities 



158 
636 



22 



4,064 
6,617 



752 



2,665 



Regular meetings 
(191 times) 
Ad hoc meetings 
(371 times) 
In-house seminars 
(7 times) 

Outpatient system 
(8 times) 
Inpatient system 
(4 times) 



825 



14,095 



2. Business Process for PHIS 

The business process for PHIS was derived from the Compo- 
nent Business Model (CBM) by IBM [ 1 1 ] . As seen in Figure 2, 
the overall hospital industry CMB map was defined by using 
154 selected hospitals, the global CBM map was categorized 
into several hospital classes according to size and owner- 
ship of the hospitals, CBM map for each hospital class was 
analyzed, and the common and individual process for PHIS 
were defined. 

3. Information Technology (IT) Infrastructure 

The IT infrastructure for PHIS is comprised of various in- 
formation technologies, as shown in Figure 3. PHIS was 
designed by component-based development and developed 
by the Rich Internet Application User Interface tool and 
medical specialized framework. Web 2.0 which allows for a 
dynamic web service was applied to the UI layer to provide 



Date 


Activity 1 


2009.03 


Designation of demonstration site for new 
public hospital information system 


2009.05 


Site visits and benchmarking of other hospital 
systems 


2009.06 


Conducting workshops and interviews for 
hospital ataffs 


2009.08 


Completing system design 


2010.01 


Opening system development center 


2010.07 


Conducting user satisfaction survey 


2010.08 


Conducting system tests 


2010.09 


Conducting user training 


2010.10- 


Conducting system rehearsals (1-8) 


2011.01.29 


Opening new PHIS 




Figure 1. History of Pusan Hospital 
Information System (PHIS) 
development. 



Global CBM 
map analysis 



Analyze mega 
proocess, process chain, 
andthe detailed 
process for 154 
selected hospitals 




Categorize 
global CBM 
map into several 
hospital classes 



Institutional CBM 
map analysis 



Analyze CBM map 
for each hospital 
class 



Common/individual 
area definition 



Define common 
and individual 
process for PHIS 



Figure 2. Development of business 
process by Component Busi- 
ness Model (CBM). PHIS: 
Pusan Hospital Information 
System. 



Vol. 17 • No. 1 • March 2011 



www.e-hir.org 69 



Hyung Hoi Kim et al 



Healthcare Informatics Research J-J 



a user-oriented business environment. In addition, Flex 
3.0, which is a well-known RIA solution by Adobe Systems 
(Mountain View, CA, USA) was used to design a quality 
screen with less network traffic for complex clinical tasks. 
With Flex 3.0, users can select a widget for designing their 
own individualized screen, and quickly search for the spe- 
cific widget by using a widget docking function. 

Live Framework 5.0, which is an application development 
tool by Hyundai Information Systems, Inc. (Seoul, Korea), 
was used to develop applications in the business layer such 
as medical, medical support, ADT, and customer relations 
management. Live Framework 5.0 is a web application fra- 
mework based on Java EE 5, which was developed by Hyun- 
dai Information Systems, Inc. for medical applications by 



using component-based infrastructure such as Web Mapper, 
Service Container, and Query Document Access Object. It 
was well integrated with Flex 3.0 to reduce complexity and 
improve efficiency and stability in system development. 

4. System Overview 

The newly developed PHIS is comprised of five subsystems: 
medical (CPOE, EMR, CDSS, nursing), medical support (lab, 
radiology, and pharmacy), administration (billing, insurance 
claims, ADT), PACS, and Customer Relations Management 
(CRM), as shown in Figure 4. These systems were developed 
by the web-base approach. The key functions of these sub- 
systems are summarized as follows: 
EMR is the core of the medical subsystem and is closely 



Business layer 



Persistence layer 



Presentation layer 




Business 
Component 

Service container 



Live framework 



Web application server 



Java EE 5 



Figure 3. Information technology in- 
frastructure for Pusan Hos- 
pital Information System. 



Administration Medical 
EMR 



Medical 
supporting 



Interface 



Security 



Figure 4. Pusan Hospital Information 
System overview. IT: infor- 
mation technology, PACS: 
Picture Archiving and Com- 
munication System, EMR: 
Electronic Medical Record, 
CRM: Customer Relations 
Management, RFID: Radio 
Frequency Identification. 
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integrated with medical support subsystems, administra- 
tion subsystems, and PACS. EMR for PHIS has the following 
functions: 

• Image editing function for anatomical regions 

• Search function for the past history of old patients 

• Environment setting function for order/patient selection/ 
button customized for patient treatment pattern 

• Nursing care recording function for diagnosis (D), nursing 
care (N), action (A), and results reporting (R) by using de- 
partmental set based on the International Classification for 
Nursing Practice (ICNP) 

• User management function for access control for users 
PACS provides digitalized medical images and stores them 

in the image database, which allows the test results collected 
from over 50 medical devices to be transmitted to outpatient 
clinics, inpatient wards, Intensive Care Unit, and emergency 
center. It is fully integrated with EMR and OCS for smooth 
information flow. 

CRM provides information to improve the quality of pa- 
tient services. It has the following functions: 

• Comprehensive patient management function for provid- 
ing single-view customer support based on information 
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from the integrated customer database 

• Campaign management function for providing customer 
information via SMS and email by templates for regions, 
age group, case-mix, and tests to support the promotion 
campaign 

• Notification function for providing information on reser- 
vations for outpatient visits and physical examination 

A mobile system is under development and will provide 
patient information through mobile phones. Duty doctors at 
the emergency center carry a radio frequency identification 
(RFID) tag so that their check-in and check-out times are 
automatically recorded. RFID tags will replace most of the 
bar codes in the future. 

A security system was developed according to the govern- 
ment security guidelines as follows: 

• Hospital network is protected from outside hacking by a 
firewall 

• Individual passwords are authenticated when they are 
stored in the database 

• Logged-in screens are automatically logged-out after a pre- 
determined time period to prevent unauthorized access 

• Every transaction is automatically logged for tracking 




Figure 5. Flow of medical templates. 
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down unauthorized access 
5. Characteristics of PHIS 

1) Creation of individual medical/ nursing templates 

PHIS allows for the provision of various customized docu- 
ments by individual setting of the computing environment. 
It also allows for easy creation of nursing records, by com- 
bining several nursing templates based on ICNP, and of in- 
dividual and departmental charts by using a combination of 
blocked charts and pasting of images (Figure 5). 

2) Integrated OCS 

PHIS enables all authorized information to be accessed by 
Single Sign In and supports one-stop services for the spe- 
cialized clinical centers. Prescription orders, test orders, 
insurance codes, and logistics codes used in OCS were fully 
integrated, which has improved the consistency in clinical 
information, reduced duplication, and improved the accu- 
racy in billing and patient statistics. 

3) Comprehensive administration functions 

PHIS allows for simultaneous registration/billing at all 
windows, automatic calculation of account receivables, 
simulation of treatment costs, and automatic confirma- 
tion of eligibility for severe/rare disease patients and of the 
remaining drug prescription days for the purpose of insur- 
ance reimbursement. It also provides various consent forms, 
test guidelines, and class schedules for various diseases to 
improve patient services. By integrating all internal systems, 
ADT staffs can monitor the availability of hospital beds and 
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the status of lab tests in real-time, and effectively manage 
antibiotics, narcotics, Total Parenteral Nutrition, and anti- 
cancer drugs. These functions help improve patient services, 
staff productivity, and the efficiency of administrative activi- 
ties (Figure 6). 

III. Results 

1. Qualitative Performance Evaluation 

1) Subjects and methods 

Questionnaire survey was conducted to identify the fac- 
tors influencing IS success in July, 2010. 41 doctors, 104 
nurses, 1 1 medical support staff, and 25 administrative staff 
responded to the survey with response rate of 79%. 14 KPIs 
were derived from three layers based in PRM. Specifically, 
the input layer was comprised of IT factors (e.g., system 
quality, information quality, user understanding of system, 
information sharing, and security) and human factors (top 
management support). The process layer was comprised of 
transaction process (efficiency of process and quality of busi- 
ness transaction) and management process (regulation man- 
agement, knowledge management, and security manage- 
ment). The business layer includes the effects of the system 
on the efficiency of business outputs and user satisfaction 
[12]. 

2) Results 

As shown in Table 2, the average input layer score was 3.16: 
user understanding of the system score was the highest (3.46) 
but the system quality score was the lowest (2.80). Specifical- 
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Figure 6. Flow of screens for administrative system. 
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ly, nursing staff were more dissatisfied with the slow response 
time than the other groups because they used the system 
more frequently. A high user understanding score may ease 
the adaptation to the new system for users at PNUH. The 
top management support score was high for administrative 
staffs because they interact with the top management more 
frequently than the other groups. 
The average process score was 3.35: improved security man- 
agement was the highest (3.45), and the efficiency in trans- 
action process was the lowest (3.20). Specifically the process 
scores for doctors and nurses in the areas of reduction in 
medication errors, regulation management, knowledge man- 
agement, and security management were lower than non- 
medical staffs, perhaps due to the weak medical functions of 
the systems. 

The average score for the business layer (3.57) was higher 
than the scores for the input and process layers, perhaps us- 
ers were well adapted to the old system despite of system 
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problems. Specifically, the average business scores for ef- 
ficiency of business outputs and user satisfaction were the 
lowest for doctors (Table 2). 

2. Economic Performance Evaluation 

1) Subjects and Methods 

Two surveys were conducted to estimate the reduction in 
outpatient waiting time by comparing the time at PNUH 
with that at S hospital, a hospital similar to PNUH except for 
its implementation of EMR. 426 and 228 outpatients were 
surveyed at PNUH and S hospital, respectively [13]. 

Economic performance of HIS was evaluated based on the 
IE approach in order to account for the intangible benefits. 
The IE approach broaden the scope of the benefits to value 
and categorize it into 5 specific value items: direct benefits, 
value acceleration, value linkage, value restructuring, and 
innovation valuation. Direct benefits refer to the traditional 



Table 2. Key Performance Indicator (KPI) scores by layers and user groups 





Category 


KPI 


Doctor 


Nurse 


Medical support 


Administration 


KPI average 


Input layer 


Information 
technology 


System quality 
(response time) 


2.91 


2.44 


2.86 


2.50 


2.80 






Information quality 


3.16 


2.97 


2.79 


3.42 


3.08 






User understanding of 


3.40 


3.40 


3.49 


3.55 


3.46 






the system 
















Security system 


3.08 


3.10 


3.24 


3.35 


3.19 




Human resources 


Top management support 


3.25 


3.16 


3.09 


3.40 


3.23 




management 
















Average 




3.18 


3.09 


3.17 


3.40 


3.16 


Process layer 




Transaction 


Reduction in personnel 


3.20 


3.33 


2.91 


3.36 


3.20 




process 


costs 
















Reduction in medication 


3.13 


3.32 


3.55 


3.51 


3.38 






errors 














Management 


Regulation management 


3.08 


3.28 


3.36 


3.60 


3.33 




process 


















Knowledge management 


3.13 


3.38 


3.64 


3.40 


3.39 






Security management 


3.29 


3.43 


3.55 


3.52 


3.45 




Average 




3.19 


3.36 


3.47 


3.49 


3.35 


Business 


Efficiency 


Timeliness of information 


3.45 


3.63 


3.82 


3.68 


3.64 


process 




retrieval 














User satisfaction 


Improvement in work 
environment 


3.20 


3.51 


3.55 


3.76 


3.50 




Average 




3.33 


3.57 


3.69 


3.72 


3.57 
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benefits from HIS such as savings in personnel costs due to 
reduced workloads. Value acceleration refers to the value 
associated with the time factor such as the reduced opportu- 
nity costs for patients due to the reduced waiting time. Value 
linkage refers to the value accrued from compounded effects 
such as savings for hospital costs due to reduced medica- 
tion errors. Value restructuring refers to the value accrued 
from the increased productivity by reengineering business 
process. Innovation of valuation refers to the added value ac- 
crued from the introduction of new systems or policies (e.g., 
information sharing among hospitals). Sensitivity analysis 
was performed to test the validity of the hypothetical pa- 
rameters used in the economic analysis and to simulate the 
effects of policy alternatives such as dissemination of infor- 
mation system for PNUH to other public hospitals. 

2) Results 

The cost item includes hardware, software, network, main- 
tenance, and depreciation. Depreciation was estimated by 
assuming 5 years of economic life and no salvage cost. Main- 
tenance costs was assumed to be free until 2012 and 10% of 
the total system costs thereafter. The annual discount rate 
was assumed to be 3.69%. 

Benefits were estimated based on the assumptions that 
only direct benefits (e.g., savings from reduced paper works) 
would be derived in the first year of implementation (2011), 
that value acceleration (e.g., savings from opportunity costs 
due to reduced waiting time), value linkage (savings from 
the reduced medication errors and improved quality of care), 
and value restructuring (savings from improved work pro- 
cess) would be derived from 2012 after users become famil- 
iar with the new systems, and that innovation of valuation 
(e.g., savings from re-examination of laboratory and X-ray 
tests) would be derived from 2013 after government policy 
allows for information sharing among hospitals. 

The cumulative benefits to costs (B/C) ratio was 0.50 in 
2011, 1.73 in 2012, 1.76 in 2013, 1.71 in 2014, and 1.71 in 
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2015. As seen in Table 3, the B/C ratios steadily increase as 
value items are added. The total B/C ratio was 0.47 if only 
the direct benefit was included, but increased to 0.86 with 
the addition of value acceleration, to 0.93 with value linkage, 
to 1.50 with value restructuring, and to 1.51 with innovation 
valuation (Table 3). 

IV. Conclusions 

The new PHIS was developed by the web-based IT infra- 
structure and began implementation on January 29, 2011. 
Before its implementation, the performance of the old PHIS 
was qualitatively evaluated based on PRM in order to iden- 
tify problem areas for the old system and collect baseline 
data for comparing with the performance of the new system. 
In the input layer, the system quality was low and nursing 
staff were more dissatisfied with the slow response time than 
other staff, perhaps due to their frequent use of the system. 
In the process layer, the accuracy of the business process was 
low and especially doctors were more dissatisfied with inac- 
curacies in the business process and program errors. Finally, 
in the business layer, the average score for the business layer 
was higher than the scores for the input and process layers. 
However, doctors were less satisfied with the system, perhaps 
due to the weak clinical function in the old systems. The 
results showed that system performance, especially response 
time, and clinical function should be improved. The newly 
developed EMR and CDSS are likely to improve the clinical 
function for doctors and nurses. 

We also performed economic analysis for the newly devel- 
oped HIS by using the IE approach and performed sensitiv- 
ity analysis for the hypothetical parameters and the scenarios 
according to the number of hospitals to be disseminated. 
We used the IE approach in order to quantify the intangible 
benefits systematically. IE attempts to categorize the ben- 
efits into five value types: direct benefits, value acceleration, 
value linkage, value restructuring, and innovation valuation. 



Table 3. Five year projection of benefits to costs ratios by benefit item 



^^^^^^ Year 
Benefit items ^^^^^^ 


2011 


2012 


2013 


2014 


2015 


Total 


Direct benefits 


0.50 


0.46 


0.46 


0.45 


0.45 


0.47 


Value acceleration 




0.95 


0.95 


0.92 


0.92 


0.86 


Value linkage 




1.04 


1.04 


1.01 


1.01 


0.93 


Value restructuring 




1.73 


1.73 


1.68 


1.68 


1.50 


Innovation valuation 






1.76 


1.71 


1.71 


1.51 
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While actual hospital data were used in most of the analysis, 
some data were extrapolated from the literature if actual data 
were not available. The patient waiting time for two outpa- 
tient departments were compared for two hospitals (one 
with and one without EMR) in order to estimate the time 
savings from the reduction in waiting time. In the results, 
the B/C ratio was 0.47 if only the direct benefit was included, 
but increased to 0.86 with the addition of value acceleration, 
to 0.93 with value linkage, to 1.50 with value restructuring, 
and to 1.51 with innovation valuation. The IE analysis dem- 
onstrated the economic profitability of the PHIS project if all 
intangible benefits were included. Sensitivity analysis showed 
that the B/C ratio was further increased if the same system 
was disseminated to other public hospitals. 

In the future, a performance of the new PHIS should be 
evaluated again after 6 months of its implementation based 
on qualitative as well as quantitative (or economic) perspec- 
tives by using the above results as baseline data. Specifically, 
KPIs from the previous survey should be compared with 
the KPIs from the second survey to determine whether new 
PHIS will indeed improve user satisfaction, especially for 
medical staffs, and improve efficiency in business process. In 
addition, economic performance of new PHIS should be an- 
alyzed again by accounting for the actual changes in waiting 
time for outpatients and transaction processing time after 6 
months of its implementation. 
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